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Substance  Abuse  and  Mental  Health  Services  Administration 
Center  for  Substance  Abuse  Prevention 


Technical  Assistance  Bulletin 


Evaluating  the  Results  of 
Communication  Programs 


O  tr> 


Prevention  program  planners  sometimes  find  evaluation  difficult  to  do,  and  out- 
come evaluation  may  seem  to  be  the  most  difficult.   This  bulletin  describes  how  to 
plan  and  conduct  outcome  evaluation  and  explains  the  benefits  of  conducting 
outcome  evaluation  to  document  the  results  of  communication  efforts. 


"% 


Evaluation  is  a  continuous  process  that 
begins  with  identifying  the  prevention 
problem  or  issue  to  be  addressed  and 
who  is  affected  by  the  problem. 

Then,  as  shown  in  the  Health  Communi- 
cations Process,  evaluation  continues 
throughout  the  development  and  imple- 
mentation of  the  communication  pro- 
gram. 

Types  of  evaluation  include  formative, 
process,  outcome,  impact,  and  efficiency. 
Each  of  these  serves  a  different  purpose 
and  requires  varying  amounts  of  techni- 
cal expertise  and  resources.  In  this 
bulletin,  these  types  of  evaluation  are 
defined  as  follows: 

Formative  evaluation  (or  formative 
research)  is  conducted  as  a  part  of 
program  development.  Collecting  evalua- 
tion data  at  this  early  point  can  help 
make  decisions  about  target  audience 
selection  and  the  types  of  communication 
messages,  channels,  and  activities  to  be 
used.  Pretesting  messages  and  materials 
to  assure  their  effectiveness  prior  to  final 
production  is  one  kind  of  formative 
evaluation. 


The  Health 
Communications  Process 

Where  does  evaluation  fit  in? 


Process  evaluation  addresses  how  the 
prevention  communication  program  is 
implemented: 

■  Is  the  program  involving  target  audi- 
ence members?  How  many? 

■  Are  the  planned  activities  being  con- 
ducted? By  whom?  By  when? 
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"The  challenge  most  threatening  to 
the  goal  of  finding  solutions  to  the 
problem  of  drug  abuse  is  the  cur- 
rent lack  of  evaluation  evidence  to 
demonstrate  the  success  of  indi- 
vidual programs.  This  failure  to 
document  results  represents  a  great 
loss  to  this  developing  field,  where 
reliable  evidence  of  success  could 
guide  so  many  efforts. " 

U.  S.  General  Accounting  Office,  1992 


Process  evaluation  data  identify  whether  the 
program  is  progressing  as  expected  so  that 
any  program  components  that  are  not  work- 
ing as  planned  can  be  adjusted.  If  the  pro- 
gram is  not  being  implemented  as  planned, 
then  the  chances  are  that  the  program  out- 
comes will  not  be  met.  This  information  is 
important  both  to  improve  the  operation  of  a 
program  and  to  replicate  it. 

Outcome  evaluation  measures  the  effects  of 
a  program  on  the  target  population  in  the 
short  term  and  provides  evidence  of  what  has 
changed  as  a  result  of  the  program.  At  its 
best,  outcome  evaluation  is  a  comparison  of 
"before"  and  "after."  For  example: 

■  How  much  did  the  youth  know  about 
tobacco  as  an  addictive  drug  before  the 
program  ("baseline  knowledge")?  What 
were  their  intentions  to  use  tobacco? 

■  What  was  their  knowledge  after  the 
program?  Was  there  a  change?  Did  their 
intentions  to  use  tobacco  change? 

Impact  evaluation  measures  the  effects  in 
the  long  term,  such  as  the  prevention 
program's  effect  on  a  community  problem  6, 
12,  and  18  months  after  cessation  of  the 
program.  Because  impact  evaluation  ad- 


dresses a  more  complex  situation  and  re- 
quires sustained  attention  and  multiple 
program  strategies  over  time,  impact  evalua- 
tion is  usually  the  most  complex  and  costly 
type  of  evaluation. 

Efficiency  evaluation  addresses  whether 
the  program  is  making  the  best  use  of  re- 
sources within  the  context  of  the  stated 
program  objectives  and  goals. 


Outcome  Evaluation 
Step  By  Step 

Planning  outcome  evaluation  of  a  communi- 
cation program  requires  several  steps. 

I.  Begin  With  Clear  and  Action-Oriented 
Communication  Program  Goals  and 
Objectives 


Appropriate  goals  describe  the  overall  change 
that  planners  expect  from  the  program.  These 
goals,  which  address  a  broad  substance  abuse 
problem  and  often  describe  an  improvement 
in  the  substance  abuse  situation  (such  as 
making  a  school  drug-free  or  postponing  the 
age  of  first  use  among  youth),  are  prevention 
goals.  Reaching  prevention  goals  often  means 
using  a  combination  of  strategies-such  as 
strengthening  drug-free  policies,  generating 
community  support  for  activities,  and  com- 
municating about  the  efforts.  Communication 
strategies  should  be  interwoven  with  other 
prevention  strategies  to  reach  the  prevention 
goals. 

Achievable  objectives  for  a  communication 
program  specifically  relate  to  what  communi- 
cations can  reasonably  be  expected  to  con- 
tribute to  resolving  the  problem.  That  is. 
objectives  for  communication  efforts  should 
contribute  to: 

■  Raising  awareness 

■  Increasing  knowledge 

■  Influencing  attitudes  and  norms 

■  Showing  benefit  of  behavior  change 


What  Outcome  Evaluation  Can  Do 


Evaluation  offers  program  planners  the  tools 
for  seeing  where  they  are  going  and  knowing 
when  they  get  there.  It  can  help  them  to: 

■  Guide  administrative  decisions. 

Evaluation  involves  making  decisions 
about  value,  worth,  and  merit.  For  ex- 
ample, will  teaching  styles  affect  the 
impact  of  the  program  on  students?  The 
answer  to  this  question  can  help  deter- 
mine the  types  of  teachers  to  hire  or 
whether  it  will  be  worthwhile  to  invest  in 
teacher  training. 

■  Refine  the  program.  Outcome  evalu- 
ation findings  can  clarify  what  has  hap- 
pened, and  what  can  realistically  be 
accomplished.  The  program  can  then  be 
reframed  and/or  its  objectives  revised 
based  on  documented  progress  made. 

■  See  concrete  results.  Evaluation  find- 
ings help  provide  reasons  for  involve- 
ment to  community  groups  and  others 
participating  in  the  program,  help  with 
recruitment,  and  bolster  staff  morale. 

■  Prove  to  funders  the  value  of  their 
investment.  This  is  important  for  pro- 
gram continuation! 


Gain  credibility.  Without  more 
formal  evaluation,  program  planners 
have  to  rely  on  anecdotal  evidence  to 
describe  what  did  or  did  not  happen. 
Evaluation  provides  objective  data  or 
documented  facts.  Anecdotes  are 
important,  too;  but  credibility  is 
earned  by  being  able  to  explain  how 
each  activity  resulted  in  change. 

Support  replication.  Outcome 
evaluation  explains  that  the  strategy  or 
program  works.  When  coupled  with 
process  evaluation  (which  shows  how 
the  program  was  implemented),  the 
steps  needed  to  replicate  the  program 
and  to  reproduce  the  results  can  be 
described. 

Advance  the  field.  It  takes  concerted 
efforts  among  many  prevention 
specialists  to  be  successful  in  reaching 
prevention  goals.  So  it  is  important  to 
identify  and  share  "what  works  and 
what  doesn't"  to  increase  the  use  of 
communication  strategies  that  prove 
successful,  and  modify  or  replace 
those  that  are  not. 


■  Reinforcing  positive  knowledge,  attitudes, 
and  behavior 

■  Demonstrating  skills 

■  Suggesting/prompting  an  action 

■  Increasing  demand  for  services 

■  Refuting  myths  and  misconceptions. 

First,  identify  what  communications  will 
contribute  to  the  broader  prevention  pro- 
gram. Then,  write  communication  objectives 
that  are: 

■  Specific  -  written  to  point  the  way  to 
action 


Attainable  -  to  increase  the  probability 
of  success 

Prioritized  -  to  direct  the  allocation  of 
resources 

Quantified  -  to  specify  how  much 
change  is  desired 

Measurable  -  to  assess  progress 

Targeted  -  to  identify  who  should 
change 

Time  specific  -  to  tell  when  the  objec- 
tive is  to  be  reached 


Six  Steps  for  Outcome 
Evaluation: 

1. 

Begin  with  clear  and  action 
oriented  communication  program 
goals  and  objectives 

2. 

Turn  the  communication  program 
outcome  objectives  into  outcome 
evaluation  questions 

3. 

Design  the  outcome  evaluation 

4. 

Gather  data 

5. 

Analyze  data  and  write  the  evalua- 
tion report 

6. 

Use  the  findings. 

■    Useful  -  to  contribute  to  accomplishing 
the  program  goals. 

Establishing  realistic,  measurable  objectives  is 
important.  Information  on  recent  trends  can 
provide  some  clues  to  what  is  possible.  For 
example,  if  negative  attitudes  toward  cocaine 
use  have  increased  by  3  percent  among  youth 
in  the  past  year,  program  planners  may 
decide  to  set  a  goal  in  the  same  range  for  the 
next  year.  Just  make  sure  that  the  change 
aimed  for  is  reasonable  -  and  remember  that 
expecting  100  percent  of  any  group  of  people 
to  change  is  not  reasonable. 

The  goals  and  objectives  for  the  Nation  set 
forth  in  the  Healthy  People  2000  report  can 
provide  some  guidance,  (as  can  those  devel- 
oped by  many  States  and  communities).  Some 
national  substance  abuse  prevention  objec- 
tives relate  to  communications,  others  to 
broader  prevention  goals  to  which  a  commu- 
nication program  can  contribute. 

Communication  program  objectives  may 
describe  what  is  expected  to  happen  or 
change  (outcome  objectives),  and  others  may 
relate  to  how  it  will  happen  (process  objec- 
tives). The  first  step  in  planning  outcome 


evaluation  is  writing  outcome  objectives,  such 
as  the  following: 

■  By  1997,  increase  the  knowledge  of  50 
percent  of  health  clinic  counseling  staff 
on  five  key  effects  of  using  crack. 

■  By  the  end  of  the  program,  increase  by  75 
percent  the  number  of  13  to  18-year-old 
girls  who  know  the  five  immediate  effects 
of  tobacco  smoking  on  their  bodies. 

■  At  the  end  of  the  presentation,  90  percent 
of  the  runaway  youth  will  be  able  to  list 
an  agency  where  a  friend  with  an  alcohol 
or  other  drug  problem  can  get  help. 

2.  Turn  the  Communication  Program 
Outcome  Objectives  Into  Outcome 
Evaluation  Questions 

The  questions  an  outcome  evaluation  is 
designed  to  answer  should  be  based  on  the 
program's  outcome  objectives.  For  example: 

Outcome  objective 

■  By  the  end  of  the  program,  increase  by  75 
percent  the  number  of  13  to  18-year-old 
girls  who  can  articulate  the  five  immediate 
effects  of  tobacco  smoking  on  their 
bodies. 

Evaluation  questions 

■  What  does  the  target  audience  currently 
know  about  the  immediate  effects  of 
smoking  tobacco?  (This  is  the  "baseline.") 

■  What  do  those  who  participate  know  after 
the  program? 

■  During  this  same  time,  what  do  nonpar- 
ticipants  within  the  target  audience  know? 

If  more  than  one  target  audience  (such  as 
youth  leaders  and  other  youth)  is  involved, 
measure  changes  with  each  group.  And  if 
others  such  as  teachers  are  involved,  add 
evaluation  questions  to  address  their  role 
(such  as  identifying  changes  in  teaching 
methods).  Before  moving  ahead  to  evaluation 
design,  make  sure  that  the  evaluation  ques- 
tions focus  on  the  most  important  issues  for 
the  program. 


Case  Study  1 

Across  Ages:  An  Intergenerational 
Mentoring  Approach  to  Drug  Preven- 
tion (Philadelphia,  PA)  matches  elder 
mentors  to  help  sixth  grade  students  in 
high-risk  environments.  Across  Ages  also 
includes  community  service  for  students, 
workshops  for  parents,  and  a  classroom 
curriculum.  Program  evaluation  includes 
both  process  and  outcome  evaluation 
components. 

The  outcome  evaluation  design  includes 
two  randomized  intervention  groups  and  a 
control  group  using  pretest  (before  the 
intervention)  and  posttest  (after  the  inter- 
vention) measures.  Within  each  school, 
three  sixth  grade  classes  (from  among 
those  with  teachers  who  agreed  to  partici- 
pate) were  assigned  to  the  control  group 
(no  program  intervention),  the  moderate 
intervention  group  (all  program  compo- 
nents except  mentoring),  or  the  full  inter- 
vention group  (all  program  components 
including  mentoring).  The  groups  were 
demographically  comparable. 

Previously  tested  evaluation  instruments 
were  used  to  measure  attitudes  toward 
alcohol  and  cigarette  use,  reactions  to 
situations  involving  drug  use,  reactions  to 


stress  or  anxiety,  self-perception,  fre- 
quency of  substance  abuse,  reactions  to 
persuasion  to  use  substances,  knowledge 
about  older  persons,  and  personal  sense 
of  well-being. 

Newly  designed  evaluation  instruments 
looked  at  attitudes  toward  school,  elders, 
and  the  future;  knowledge  about  sub- 
stance abuse;  and  problem-solving  skills. 
Also  measured  was  the  extent  to  which 
students  actively  participated  in  the 
activities  (to  determine  how  much  each 
student  was  exposed  to  the  program). 

Of  the  13  measures,  evaluators  found 
very  good  results  in  the  pretest  for  four  in 
both  the  experimental  and  control 
groups.  For  example,  students  already 
had  the  desired  attitudes  toward  drug 
use.  Therefore,  less  change  could  be 
expected  in  these  areas.  For  the  other 
nine  measures,  however,  there  was 
significantly  more  improvement  among 
students  in  the  experimental  groups  than 
students  in  the  control  group,  with  most 
improvement  among  students  who 
received  the  full  intervention.  The  use  of 
a  control  group  permitted  comparisons 
and  demonstrated  the  positive  effects  of 
both  classroom  interventions  and 
mentoring. 


3.  Design  the  Outcome  Evaluation 

There  are  various  research  designs  that  can 
be  used  for  outcome  evaluation.  Basic  text- 
books on  evaluation  provide  detailed  expla- 
nations of  these.  However,  it  may  be  useful  to 
work  with  an  experienced  evaluator  who  can 
suggest  which  research  design  is  best  applied 
under  which  conditions. 

The  best  outcome  evaluation  uses  a  random- 
ized "experimental"  design.  This  means  that 
individuals  within  the  target  audience  are 
assigned  into  two  equivalent  groups:  one 
group  will  be  a  part  of  the  program  (the 


experimental  or  intervention  group),  and  the 
other  will  not  (the  control  group). 

Ask  the  same  set  of  evaluation  questions  to 
both  groups  before  the  program  begins. 
Answers  to  these  questions  (e.g.,  on  the 
knowledge  or  attitudes  the  program  wants  to 
affect)  serve  as  the  baseline,  or  starting  point. 

Then,  ask  both  groups  the  same  questions 
after  the  program  is  finished.  Presumably,  any 
significant  differences  in  the  responses  from 
the  intervention  group,  as  compared  with  the 
control  group,  can  be  attributed  to  the  pro- 
gram. Randomized  experimental  designs 


Outcome  Evaluation  Options  To  Match  Resources 


Available  resources,  experience,  and 
expertise  all  help  guide  program  planners 
to  the  most  practical  evaluation  design  for 
their  program.  The  rule  of  thumb  for  well- 
planned  intervention  programs  is  to 
allocate  at  least  15  percent  of  the  program 
budget  to  evaluation. 

Some  type  of  outcome  evaluation,  how- 
ever, is  possible  for  almost  any  budget: 

■    Minimal  resources.  Activity  assess- 
ments (numbers  of  people  who 
participate  and  their  responses  to 
questions  about  what  they  learned, 
thought,  and/or  did  as  a  result  of  their 
involvement). 


Modest  Resources.  Pre-  and 
postassessments  of  desired  changes  in 
program  participants  (self-reported  or 
observed,  questionnaires  filled  out  by 
participants,  telephone  or  in-person 
interviews,  third-party  observation/ 
assessment);  for  media  programs,  a 
media  content  analysis  by  monitoring 
and  analysis  of  coverage  (content  and 
quantity)  appearing  in  the  media 
before/during/after  the  program. 

Substantial  resources.  Use  of  experi- 
mental design  with  intervention  and 
control  groups,  pre-  and 
postassessment  to  measure  change  after 
the  program  exposure. 


provide  the  best  evidence  of  outcomes.  It  is 
the  kind  of  outcome  evaluation  that  is  most 
acceptable  to  people  looking  for  "hard 
evidence"  of  success.  Using  an  experimental 
design  will  also: 

■  isolate  outcomes  to  determine  if  certain 
changes,  when  controlled  for,  affect  the 
result 

■  minimize  or  prevent  conditions  that 
obscure  clear  interpretation  of  results 

■  address  potential  problems  and  biases 
that  could  become  barriers  to  understand- 
ing whether  the  program  is  responsible 
for  the  outcomes. 

However,  for  some  community  communica- 
tion programs,  this  type  of  evaluation  be- 
comes practical  only  if  evaluation  expertise 
can  be  accessed. 

Other  types  of  designs  include: 

■  Collecting  information  pre-  and  post- 
intervention  (the  activity  or  program)  only 
from  participants  to  see  what  happened  to 
them  as  a  result  of  the  program.  (This 


works  best  for  short  programs,  such  as  a 
training  session,  where  program  planners 
can  be  reasonably  sure  that  the  program 
caused  or  contributed  to  the  change.) 

■  Looking  for  sources  of  existing  baseline 
information  and  conducting  postinter- 
vention  data  collection  only  (for  example, 
using  existing  community  or  State  surveys 
of  knowledge  and/or  attitudes  toward 
substance  use/abuse). 

■  Sorting  other  information  already  gath- 
ered from  participants  and  nonpartici- 
pants  (such  as  classroom  attendance, 
school  grades). 

■  Collecting  information  only  after  the 
intervention  to  see  if  participants  are 
where  program  planners  want  them  to  be 
(for  example,  ready  to  be  peer  counselors 
to  others).  This  does  not  tell  planners  if 
participants  obtained  that  status  as  a  result 
of  the  program. 

Most  outcome  evaluation  methods  are  based 
on  collecting  data  about  participants  through 
a  questionnaire,  interview,  or  observation. 


Case  Study  2 

Right  Turns  Only!  (Prince  Georges 
County,  MD)  is  a  video-based  drug 
education  series  produced  by  the  Prince 
Georges  County  School  System  with 
CSAP  funding.  The  effect  of  this  video 
series  (including  collateral  print  material) 
on  student  knowledge,  attitudes,  and 
behavioral  intentions  was  tested  at  12 
schools  among  approximately  1,000 
seventh  grade  students  assigned  to  four 
groups.  One  group  received  only  the 
video-based  education;  a  second  group 
received  the  program  in  addition  to  a 
traditional  drug  education  curriculum;  a 
third  group  received  only  the  traditional 
curriculum;  and  a  fourth  group  -  the 
control  group  -  received  no  drug  abuse 
prevention  education  during  the  time  of 
the  study.  All  interventions  were  com- 
pleted within  a  3-  week  period. 

Outcomes  measured  included  knowledge 
of  substance  abuse  terminology,  ability  to 


assess  advertisements  critically,  perception 
of  family,  conflict  resolution,  self-efficacy 
in  peer  relationships,  and  behavioral 
intentions  related  to  substance  use/abuse 
prevention.  Changes  were  measured  using 
a  questionnaire  completed  by  students 
before  and  after  the  interventions,  and  the 
questionnaires  were  analyzed  to  identify 
any  differences  based  on  gender,  race, 
grades  (self-reported),  and  teacher. 

As  hypothesized,  groups  receiving  drug 
education  scored  higher  on  all  measures 
except  self-esteem  in  the  posttest  than  did 
the  control  group.  On  two  of  the  seven 
measures,  the  group  receiving  the  video 
series  and  traditional  curriculum  scored 
significantly  higher  than  other  groups. 
Thus  evaluators  were  able  to  demonstrate 
that  instructional  television  (in  particular 
when  used  in  conjunction  with  print 
materials  and  teacher  guidance)  can  be  an 
effective  tool  for  delivering  drug  education 
in  the  classroom. 


The  type  of  data  collection  method  selected  is 
based  on  how  to  best  answer  the  evaluation 
questions,  access  to  the  target  audience,  and 
resources.  (See  the  case  studies  included  in 
this  bulletin  for  ideas.) 

Some  of  the  decisions  made  in  designing  an 
outcome  evaluation  include  what  data  will  be 
collected,  from  whom,  how,  and  when.  These 
data  help  answer  the  program's  evaluation 
questions. 

Remember  that  the  more  complex  the  evalua- 
tion design,  the  more  expert  assistance  may 
be  needed  to  design  and  conduct  the  evalua- 
tion. 

If  there  is  no  evaluator  on  staff,  seek  help  to 
decide  what  type  of  outcome  evaluation  will 
best  serve  your  program.  Sources  of  evalua- 


tion consultation  and  hands-on  assistance 
include  university  faculty,  graduate  students 
(for  data  collection  and  analysis),  local  mar- 
keters and  other  businesses  (staff,  computer 
time),  State  and  local  health  and  social  service 
agencies,  and  organizations  with  experience 
in  evaluation. 

What  Data 

The  data  collected  and  the  measures  used  to 
determine  whether  a  change  has  occurred 
should  directly  relate  to  the  evaluation  ques- 
tions. Specific  questionnaires  or  other  data 
collection  instruments  should  be  compared  to 
the  questions  they  intend  to  ask  with  the 
evaluation  questions  (Step  2)  and  outcome 
objectives  (Step  1).  For  example,  if  the  pro- 
gram is  intended  to  change  knowledge,  all  of 
the  measures  should  be  knowledge  related. 


Ensure  the  validity  of  the  data  collection 
instruments  used.  Valid  instruments  should 
measure  what  they  are  supposed  to  measure. 

Sources  for  help  in  developing  a  question- 
naire include  examples  used  by  other  pro- 
grams, evaluation  reports  from  NCADI  [see 
Resources],  evaluation  design  manuals,  and 
experienced  staff  from  other  prevention 
programs  or  university  graduate  programs. 
Such  sources  can  provide  guidance  on  how  to 
develop  data  collection  measures  and  instru- 
ments that  are  reliable  and  valid. 

If  collecting  baseline  data  is  beyond  existing 
capability,  check  for  available  data.  Has 
someone  already  collected  this  information? 
If  it  was  not  for  the  program's  geographic 
area,  is  it  reasonable  to  use  it  as  an  approxi- 
mation of  the  baseline  measures? 

From  Whom 

Think  about  from  whom  information  should 
be  gathered.  Different  data  collection  instru- 
ments and  methods  may  be  needed  for  each 
group.  For  example,  if  one  or  more  interven- 
tion groups  and  a  control  group  are  being 
followed,  each  may  need  to  be  asked  the 
same  questions  in  a  different  way  in  order  to 
get  at  a  common  construct  or  concept.  Or,  if 
more  than  one  culture  is  involved,  different 
methods  to  capture  adequately  the  experi- 
ences of  each  culture  may  be  needed. 

How 

Data  can  be  collected  in  many  ways;  for 
example,  through  qualitative  methods  such  as 
focus  groups,  open  ended  questionnaires, 
personal  interviews,  and  observation  or 
through  quanitative  methods  such  as  struc- 
tured surveys  and  questionnaires. 

Deciding  how  data  will  be  collected  is  an- 
other resource  question.  Does  the  program 
have  access  to  skilled  interviewers? 

Consider  how  comfortable  the  participants 
will  be  with  the  planned  means  of  collecting 
data.  Will  they  be  willing  and  able  to  fill  out 
the  forms?  Will  they  be  willing  to  provide 


"Positive  results  from  an  outcome  evalu- 
ation— showing  more  favorable  results 
from  participants  than  similar  nonpar- 
ticipants — offer  hard,  objective  evidence 
that  a  social  program  truly  makes  a 
difference  and  thus  is  a  positive  invest- 
ment of  human  capital. " 

U.  S.  General  Accounting  Office,  1992. 


personal  information  to  interviewers?  The 
answers  may  vary  according  to  age,  culture, 
levels  of  literacy,  and  setting,  so  pilot  testing 
data  collection  instruments  is  vital. 

When:  The  Outcome  Evaluation  Plan 

The  decisions  made  about  what  data  to 
collect,  from  whom,  and  how  are  all  con- 
nected. Completing  the  Worksheet:  Outcome 
Evaluation  Plan  (see  page  13)  will  help 
program  planners  think  through  their  evalua- 
tion plan.  Developing  a  timeline  and  task 
schedule  is  crucial.  The  collection  of  prepro- 
gram-intervention  data  must  be  completed 
before  the  prevention  communication  pro- 
gram starts.  The  more  complex  the  evaluation 
design,  the  more  important  the  timeline  and 
task  schedule  become. 

When  program  planners  complete  the 
worksheet,  it  provides  a  written  guide  to  the 
outcome  evaluation  plans.  The  plan  also  can 
help  identify  where  there  is  a  need  for  evalu- 
ation help. 

The  design  of  process  evaluation  is  not  the 
focus  in  this  bulletin,  but  process  measures 
are  important  for  any  program  conducting 
outcome  evaluation.  Information  from  pro- 
cess measures  will  help  identify  parts  of  the 
program  that  work,  parts  that  did  not  work, 
and  the  reasons  why  the  outcome  was  or  was 
not  positive. 


4.  Gather  Data 

Think  through  the  logistical  issues  involved  to 
make  sure  that  all  of  the  information  (data) 
needed  is  actually  gathered  from  the  partici- 
pants. It  is  important  to  ensure  that  data 
gatherers  are  well-trained  so  that  they  do  their 
job  thoroughly.  Uniform  data  collection 
strategies  should  be  employed  throughout  the 
process. 

5.  Analyze  Data  and  Write  the 
Evaluation  Report 

Once  information  is  gathered,  it  should  be 
analyzed  according  to  a  carefully  thought  out 
approach  (analysis  plan). 

"Analysis"  means  looking  methodically  at  the 
information  and  determining  what  answers  it 
provides  to  the  evaluation  questions.  It  is  also 
important  to  write  down  any  reasons  why  the 
program  did  or  did  not  work  as  expected,  and 
what  should  be  changed  or  evaluated  differ- 
ently in  the  future. 

If  this  process  is  new  to  program  planners, 
they  may  need  the  help  of  an  evaluation 
specialist  to  guide  them  through  data  collec- 
tions and  analysis. 

Once  analysis  is  complete,  think  about  how 
the  evaluation  report  will  be  used  before 
writing  it.  And  as  with  all  communication 
materials,  it  is  important  to  consider  who  the 
readers  (target  audiences)  will  be.  A  report 
prepared  for  journal  publication  may  look 
very  different  from  one  shared  with  the 
community. 

Consider: 

■  Preparing  different  versions  for  different 
readers  and  purposes 

■  Using  visuals  to  highlight  important 
findings 

■  Incorporating  information  from  other 
studies,  programs,  or  sources  to  add 
weight  to  findings  (such  as  national  data 
that  are  comparable,  other  studies  that 
came  to  similar  conclusions). 


6.  Use  the  Findings 

An  evaluation  report  that  resides  in  the  office 
bookcase  is  not  supporting  the  program.  The 
distribution  plan  might  include: 

■  Project  staff  (for  discussion  about  implica- 
tions for  the  program) 

■  Managers  and  other  influential  people  in 
the  agency  or  organization 

■  Funding  sources  (outcome  data  may  be 
crucial  to  further  funding) 

■  Other  citizens  groups,  agencies,  and 
businesses  that  were  involved  in  the 
program 

■  Others  that  program  planners  might  want 
to  work  with  in  the  future 

■  Local  or  State  legislators 

■  Prevention  specialists  and  similar  pro- 
grams in  other  locations 

■  NCADI  (see  Resources). 


Cultural  Competence  in 
Evaluation 

Planning  for  prevention  communication 
programs  requires  forming  a  set  of  assump- 
tions about  what  should  happen  to  contribute 
most  effectively  to  substance  use  prevention, 
who  should  be  involved  or  affected,  and  what 
results  may  be  expected.  These  assumptions 
must  take  into  consideration  the  norms, 
values,  and  expectations  of  the  community 
and  its  cultural/ethnic  groups.  Among  these 
different  groups,  for  example,  norms  con- 
cerning who  is  in  the  best  position  to  influ- 
ence youth  and  to  lead  the  community  may 
vary. 

Planning  for  evaluation  must  also  take  cul- 
tural considerations  into  account,  particularly 
in  selecting  methods  for  gathering  informa- 
tion and  in  interpreting  the  results.  Consider, 
for  example,  that: 


Case  Study  3 

La  Esperanza  del  Valle  (Yakima  Valley, 
WA)  is  a  multimedia  prevention  soap 
opera  campaign  targeted  to  Latino  adoles- 
cents and  families  at  risk  that  was  devel- 
oped by  Novela  Health  Education  and  the 
University  of  Washington  Health  Educa- 
tion and  Training  Center.  The  video 
component  was  shown  to  students  in  the 
classroom.  To  evaluate  the  efficacy  of  the 
video  in  changing  attitudes  and  behaviors 
related  to  alcohol  use,  students  com- 
pleted a  pretest  1  to  2  weeks  before  the 
viewing  and  a  posttest  form  within  1 


week  after  seeing  the  video.  For  students 
who  understood  enough  spoken  English 
to  understand  the  video  but  did  not  have 
sufficient  skills  to  read  the  questionnaire 
in  English,  the  questions  were  read  in 
English  and  repeated  in  Spanish. 

The  evaluation  identified  some  positive 
changes  in  youth  attitudes  regarding 
alcohol  from  pre-  to  post-viewing,  includ- 
ing significant  changes  in  intentions  to 
drink.  The  majority  (74%)  of  responding 
students  indicated  that  if  they  used 
alcohol  they  would  probably  or  definitely 
change  their  drinking  behavior  based  on 
what  was  shown  in  the  video. 


In  some  cultures  speaking  out  in  a  group 
(such  as  a  focus  group)  or  revealing 
negative  feelings  about  an  activity  is  not 
appropriate.  Thus  observing  nonverbal 
cues  may  be  more  revealing  than  oral 
communications . 

Some  groups  distrust  or  fear  providing 
information  to  a  person  from  a  different 
culture.  Thus  the  ethnicity  of  the  data 
gatherer  may  inadvertently  influence  the 
information  provided. 

Some  groups  may  lack  familiarity  with 
how  to  fill  out  a  printed  questionnaire,  or 
they  may  face  difficulties  due  to  limited 
English  skills.  The  Asian  Drug  and  Alco- 
hol Prevention  Team  (Utah)  found  it 
difficult  to  use  pre-  and  posttest  instru- 
ments with  an  Asian  student  population 
living  in  a  high-risk  environment:  "In 
most  cases,  the  problem  was  limited 
English;  in  other  cases,  participants  simply 
were  not  familiar  with  the  testing  proce- 
dures or  the  concepts...  we  were  attempt- 
ing to  gauge."  The  team  decided  instead 
to  measure  outcomes  in  ways  that  in- 


cluded school  attendance  and  progress, 
change  in  family/individual  behavior,  and 
nonverbal,  self-reported  measures. 

The  cultural  outlook  of  the  evaluator  may 
inadvertently  affect  the  objectivity  of 
evaluation  reports  on  program  activities. 
Thus  finding  a  member  of  the  community 
to  conduct  the  evaluation  or  making  sure 
to  involve  a  culturally  competent  evalua- 
tor may  be  important. 

Programs  that  cut  across  cultures  and 
adapt  their  evaluation  methods  to  fit 
different  groups  may  find  it  difficult  to 
compare  results  across  groups.  Thus  these 
types  of  evaluations  are  more  complicated 
and  usually  require  more  evaluation 
expertise. 

If  evaluation  materials  are  translated  into 
another  language,  have  them  translated 
back  to  the  original  prior  to  use.  This  will 
ensure  that  the  original  meaning  has  not 
been  lost. 


• 
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Outcome  Evaluation  Myths  and  Misconceptions 


If  the  outcome  from  an  activity  is 
good,  it  is  a  good  program.  Having 
positive  results  is  definitely  a  plus,  but 
the  results  must  contribute  to  meeting 
program  objectives.  Sometimes  it  is  easy 
to  get  carried  away  by  a  "good  idea"  for  a 
communication  activity  or  a  creative 
slogan.  Make  sure  that  the  good  idea  will 
move  the  program  a  step  closer  to  the 
outcome. 

The  program  does  not  have  a  lot  of 
resources,  so  counting  is  sufficient. 

Numbers  can  be  impressive.  But  anyone 
looking  at  the  program  with  a  critical  eye 
is  likely  to  ask  the  next  question:  "But 
what  happened  as  a  result?"  A  communi- 
cation program  can  include  ways  to 
answer  that  question. 

Outcome  evaluation  comes  last;  it 
can  be  dealt  with  once  the  program 
gets  rolling.  That  is  too  late!  It  is  tempt- 
ing to  wait  when  other  tasks  seem  more 
immediate,  but  if  there  is  no  assessment 
of  where  you  began,  how  will  program 
planners  know  whether  their  program 
has  made  a  difference? 

Evaluation  is  too  difficult/expen- 
sive, so  it  cannot  be  done.  It  is  true 
that  evaluation  requires  an  effort.  But 
don't  program  planners  and  managers 
have  an  obligation  to  prove  that  they 
have  done  it  well?  There  are  many 
sources  of  help  to  design  and  gather 
information. 

The  budget  is  tight.  It  is  more  im- 
portant to  do  the  program,  so 
evaluation  can  be  eliminated  More 
important?  Not  if  the  program  is  to  be 
valued  in  the  community... and  continue! 


Of  course,  keeping  an  appropriate 
balance  between  resources  used  for 
program  and  evaluation  is  important; 
and  the  program  should  get  the  bulk 
of  what  is  available.  Nevertheless, 
evaluation  is  an  essential  part  of  being 
accountable. 

Evaluation  results  must  show  a 
positive  effect  to  be  useful  All 

program  planners  can  learn  from  their 
experiences,  both  from  the  positive 
and  the  negative. 

When  the  evaluation  is  complete, 
it  is  time  to  put  the  project  on 
the  shelf.  When  the  evaluation  is 
done,  it  is  time  to  tell  people  what  has 
been  learned.  Evaluation  results  are 
not  valuable  until  they  are  used. 

This  is  not  going  to  be  a  long- 
term  project,  so  outcome  evalua- 
tion is  not  important.  Are  there 
going  to  be  no  more  programs?  The 
lessons  learned  from  outcome  evalua- 
tion will  be  useful  next  time  in  devel- 
oping substance  use/abuse  preven- 
tion programs  in  your  community  and 
others. 

Agreeing  on  the  specific  out- 
comes is  too  difficult.  Agreeing  on 
what  is  to  be  accomplished,  before 
deciding  what  is  to  be  done  and 
making  sure  that  all  activities  contrib- 
ute to  reaching  the  objectives,  is 
essential. 
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Glossary 

Baseline  study.  The  collection  of  data  re- 
garding a  target  audience  or  situation  prior  to 
the  intervention. 

Control  groups.  In  research,  those  individu- 
als to  whom  no  message  or  information  is 
given,  but  who  resemble  members  of  the  ex- 
perimental group  in  all  respects. 

Efficiency  evaluation.    Research  that  ad- 
dresses how  well  a  program  is  using  its  re- 
sources within  the  context  of  the  stated 
program  goals  and  objectives. 

Experiment.  A  research  investigation  in 
which  the  investigator  has  control  over  the 
levels  of  the  independent  variable  and  over 
assignment  of  individuals  to  different  condi- 
tions (such  as  program  activities). 

Formative  evaluation.  Evaluation  research 
conducted  during  program  development.  May 
include  state-of-the-art  reviews,  pretesting 
messages  and  materials,  and  pilot  testing  a 
program  on  a  small  scale  before  full  imple- 
mentation. 

Impact  evaluation.  Evaluation  that  measures 
whether  and  to  what  extent  a  program  accom- 
plished its  stated  goals  in  the  long  term. 

Intervention.  A  type  of  study  in  which  sub- 
jects are  exposed  to  a  specified  program  (treat- 
ment) and  the  effect  evaluated  (measured). 


Methods.  The  process  and  techniques  used 
to  conduct  a  study — including  data  collection 
techniques,  subjects  (individuals)  studied,  ex- 
perimental design,  sampling  procedures,  and 
instrument  construction  and  use. 

Outcome  evaluation.  Evaluation  that  mea- 
sures the  effects  on  the  target  audience  in  the 
short  term  and  provides  evidence  of  what  has 
changed  as  a  result  of  the  program. 

Process  evaluation.  Evaluation  to  study  the 
functioning  of  the  program;  includes  assess- 
ments of  whether  materials  are  being  distrib- 
uted to  the  right  people  and  in  what 
quantities,  whether  and  to  what  extent  pro- 
gram activities  are  occurring,  and  other  mea- 
sures of  how  and  how  well  the  program  is 
working. 

Qualitative  research.  Usually,  research  that 
involves  obtaining  information  from  small 
numbers  of  people.  The  information  gathered 
is  subjective  and  not  described  in  numerical 
terms;  therefore,  generalizations  should  not  be 
made.  Can  be  very  useful  in  clarifying  issues 
for  subsequent  quantitative  research. 

Quantitative  research.  Research  designed 
to  gather  information  from  representative  ran- 
dom samples  of  respondents.  Results  are  ex- 
pressed in  numerical  terms.  Quantitative  data 
are  used  to  draw  objective  conclusions  about 
the  target  audience. 


"Cultural  competence:  A  set  of  skills  that  allows  individuals  to 
increase  their  understanding  and  appreciation  of  cultural  dif- 
ferences and  similarities  within,  among,  and  between  groups. 
This  requires  a  willingness  and  ability  to  draw  on  community- 
based  values,  traditions,  and  customs  and  to  work  with  knowl- 
edgeable persons  of  and  from  the  community  in  developing 
focused  interventions,  communications,  and  other  supports." 

Center  for  Substance  Abuse  Prevention.  1992. 
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WORKSHEET 

Outcome  Evaluation  Plan 


Problem/issue  to  be  addressed: 


Target  audience  (consider  a  separate  plan  for  each  target  audience): 


Program  goals  and  objectives  (what  the  program  is  intended  to  accomplish): 


Types  of  evaluation  to  be  conducted  (process  as  well  as  outcome): 


Evaluation  questions  to  be  answered: 


Data  collection  methods  for  each  type  of  evaluation  (e.g.,  telephone  surveys,  participant  evalua- 
tion forms  in  the  classroom): 


Analysis  plan  (what program  planners  are  going  to  do  with  the  data  gathered  in  order  to  an- 
swer the  evaluation  questions  posed): 


Evaluation  products  and  use  of  evaluation  data  (e.g.,  to  revise  program,  to  share  with  others): 


continued  on  page  14 
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WORKSHEET 

Outcome  Evaluation  Plan  continued 

Sample  Timeline  and  Task  Schedule 
Task  Person  Responsible  Timeline 


a.    Design  the  process  and 
outcome  evaluation 


b.  Identify  participants 

c.  Identify  control  group 

d.  Design/pilot  test 
questionnaires 


e.    Pretest  participants  and 
control  group 


f.     Implement  program 


g.    Posttest  participants  and 
control  group 


h.    Analyze  data 

i.     Write  evaluation  report 

j.     Share  findings 
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